{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "fsharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "#r \"nuget: TorchSharp-cpu\"\n",
    "\n",
    "open TorchSharp\n",
    "open type TorchSharp.torch\n",
    "open type TorchSharp.TensorExtensionMethods"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Numerics\n",
    "\n",
    "Arithmetic is what TorchSharp is all about, and the capabilities are rich. It's all about tensor arithmetic, though -- that's where GPU acceleration makes sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "fsharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "let a = torch.ones(3,4)\n",
    "let b = torch.zeros(3,4)\n",
    "let c = torch.tensor(5)\n",
    "a * c + b"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's often the case that you can reuse the storage for one of the operands, so TorchSharp defines a number of 'in place' operators. These will only work if the operand has the same shape and layout as the result, of course. To use the in-place operators, you can't use the nice math syntax, you have to use functions. TorchSharp follows the PyTorch convention of appending a '_' to the name of in-place operators. It's very similar to the '*=', '+=', etc. operators in C#, except that they can be chained together.\n",
    "\n",
    "In the expression below, the storage for 'a' is used to hold first the result of multiplying with c, and then adding b."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "fsharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "a.mul_(c).add_(b)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After this, 'a' is not longer holding ones, since it's been overwritten. The impact on performance that using in-place operators has is significant, if used consistently, but it's important to know what you're overwriting and not to over-use in-place operators. Think of it as a performance optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "fsharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "a"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Broadcasting"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the simple example above, you saw that 'c' was defined from a single value. If we look at it, we can see that it's a singleton tensor. That is, it has no shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "fsharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "c.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "fsharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "c"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though its shape differed from that of 'a,' we were able to use it in the computation. How come?\n",
    "\n",
    "TorchSharp will adjust the shape, without allocating new memory, of a tensor to be compatible with another tensor in situations like this. This is called 'broadcasting' and is found in most every numerics and deep learning library around. It's not just singletons that can be broadcast -- any tensor that is compatible will have it work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "fsharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "let a = torch.ones(3,4)\n",
    "(a + torch.ones(4)).print()\n",
    "a + torch.ones(1,4)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Numerics Library\n",
    "\n",
    "The collection of numerical operators that are available is too large to go through here, but suffice it to say that all the usual suspects are available. Most of the operate on an element-wise basis, i.e. the operator is applied to each element of the operands, possibly with broadcasting getting involved.\n",
    "\n",
    "One notable and __very__ significant exception is matrix multiplication, which is vector dot product generalized to matrices. The '*' operator denotes element-wise multiplication, while matrix multiplication is performed by the 'mm' method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "fsharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "let a = torch.full(4L,4L, (17).ToScalar())\n",
    "let b = torch.full(4L,4L, (12).ToScalar())\n",
    "\n",
    "(a * b).print()\n",
    "(a.mm(b))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some very specialized operators doing more than one thing at a time avoiding creating temporaries. Some of them are there because the absence of temporaries can lead to more numerical stability (such as avoiding rounding error propagation), or because you don't have to go back and forth between the CPU and GPU as often. It is almost always the right choice to use these special composite operators when they are a match for your computation.\n",
    "\n",
    "An example is xlogy(), which performs x * log(y) all in one operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "fsharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "let x = torch.rand(5)\n",
    "let y = torch.rand(5)\n",
    "(x * torch.log(y)).print()\n",
    "x.xlogy(y)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".NET (F#)",
   "language": "F#",
   "name": ".net-fsharp"
  },
  "language_info": {
   "name": "F#"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
